Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning

Recently, research on accelerated stochastic gradient descentmethods (e.g., SVRG) has made exciting progress (e.g., lin-ear convergence for strongly convex problems). However,the best-known methods (e.g., Katyusha) requires at leasttwo auxiliary variables and two momentum parameters. Inthis paper, we propose a fast stochastic variance reductiongradient (FSVRG) method, in which we design a novel up-date rule with the Nesterov’s momentum and incorporatethe technique of growing epoch size. FSVRG has only oneauxiliary variable and one momentum weight, and thus itis much simpler and has much lower per-iteration complex-ity. We prove that FSVRG achieves linear convergence forstrongly convex problems and the optimal O(1/T ) conver-gence rate for non-strongly convex problems, where T is thenumber of outer-iterations. We also extend FSVRG to di-rectly solve the problems with non-smooth component func-tions, such as SVM. Finally, we empirically study the per-formance of FSVRG for solving various machine learningproblems such as logistic regression, ridge regression, Lassoand SVM. Our results show that FSVRG outperforms thestate-of-the-art stochastic methods, including Katyusha. KeywordsStochastic optimization, variance reduction, momentum ac-celeration, non-strongly convex, non-smooth